Support glint for remote eye tracking

ABSTRACT

Methods and corresponding systems of controlling illuminators in an eye tracking system are disclosed. The system includes a first image sensor, a second image sensor, a first close illuminator arranged to capture bright pupil images by the first image sensor, a second close illuminator arranged to capture bright pupil images by the second image sensor and one or more far illuminators arranged to capture dark pupil images by the first image sensor and the second image sensor. In the methods main and support illuminators are controlled during exposure of a first and a second image sensor to produce enhanced contrast and glint position for eye/gaze tracking.

CROSS-REFERENCE TO RELATED APPLICATION

The present application claims benefit to U.S. provisional patentapplication Ser. No. 62/649,390, filed on Mar. 28, 2018, to Sjöstrand etal., entitled “SUPPORT GLINT FOR REMOTE EYE TRACKING”, and is herebyincorporated by reference in its entirety.

TECHNICAL FIELD

The present disclosure generally relates to the field of eye tracking.In particular, the present disclosure relates to systems and methods forcontrolling illuminators in an eye tracking system.

BACKGROUND

In eye tracking applications, digital images of the eyes of a user arecaptured by one or more cameras/image sensors and the digital images areanalyzed in order to estimate an eye position and a gaze vector of theuser. There are different methods for achieving such an estimation. Insome methods ambient light is used when retrieving images of the eyes ofthe user and in some methods additional light sources (illuminators) areused to illuminate the eyes for retrieving images of the eyes of theuser. Generally, the estimation of the eye position and the gaze vectoris based on identification of the pupils of the eyes of the user,together with identification of glints (corneal reflections) in the eyesof the user. In order to identify a pupil of an eye in an image of theeye, the contrast between a representation of the pupil and arepresentation of an iris of the eye in the image must be sufficient.

One known method of eye tracking includes the use of infrared light andan image sensor. The infrared light is directed towards the eye of auser and the reflection of the light is captured by an image sensor toproduce an image of the eye. Through analysis of the captured image, theposition of the eye in space and the point of gaze is estimated. Onesuch system is described in U.S. Pat. No. 7,572,008 (which is herebyincorporated by reference in its entirety).

When estimating an eye position and a gaze vector based onidentification of the pupils of the eyes of the user, together withidentification of glints (corneal reflections) in the eyes of the user,the contrast between a representation of the pupil and a representationof an iris of the eye in the image must be sufficient and the glintsshould be identified and preferably be positioned on a part of thecornea which may be approximated as spherical.

There are two different options of producing contrast between therepresentation of the pupil and the representation of the iris in animage of the eye. Either an illuminator (called close illuminator in thefollowing) is used arranged coaxially with (or close to) the lightsensor, or an illuminator (called far illuminator in the following) isused which is arranged non-coaxially with (or further away from) thelight sensor. For the close illuminator, the light sensor may capturebright pupil (BP) images of the user's eyes. Due to the coaxialarrangement of the close illuminator and the light sensor, lightreflected from the retina of an eye returns back out through the pupiltowards the light sensor, so that the representation of the pupilappears brighter than the representation of the iris surrounding it inimages captured with only the close illuminator illuminating the eye.For the far illuminator, the light sensor may capture dark pupil (DP)images. Due to the non-coaxial arrangement of the far illuminator andthe light sensor, light reflected from the retina of an eye does notreach the light sensor and the representation of the pupil appearsdarker than the representation of the iris surrounding it in imagescaptured with only the far illuminator illuminating the eye.

Whether a system for eye/gaze tracking captures BP or DP images dependson the user/subject. Some users have a high (good) BP response whichcauses the representation of the pupil to become sufficiently bright inrelation to the representation of the iris surrounding it in the image.For such users, the BP images will generally provide the best contrastbetween the representation of the pupil and the representation of theiris in the image for determining the position the representation of thepupil in the image. Some users have a low (bad) BP response which causesthe representation of the pupil not to be sufficiently bright inrelation to the representation of the iris surrounding it in the image.For such users, the DP images will generally provide the best contrastbetween the representation of the pupil and the representation of theiris in the image for determining the position the representation of thepupil in the image.

Accuracy and precision are used as indicators of the eye tracker datavalidity. A system with good accuracy and precision will provide morevalid data as it is able to truthfully describe the location of aperson's gaze on a screen. Accuracy is defined as the average differencebetween the real stimuli position and the measured gaze position.Precision is defined as the ability of the eye tracker to reliablyreproduce the same gaze point measurement, i.e. it measures thevariation of the recorded data via the Root Mean Square (RMS) ofsuccessive samples.

To enhance the precision and accuracy of eye tracking a system can beprovided with two cameras/image sensors. Each of the image sensorscaptures an image of an eye and the two images thus captured are used toenhance the precision and accuracy of eye/gaze tracking. By using twocameras, better precision and accuracy can be achieved even foridentification of only one glint. For system using one camera, generallyidentification of two glints are required.

To encompass both the BP and DP images, the system is provided with aclose illuminator for each of the image sensors and one or more farilluminators for the image sensors.

For the BP case, an image is first captured by one of the image sensorswhen the eye is illuminated by its associated close illuminator. Then animage is captured by the other one of the image sensors when the eye isilluminated by its associated close illuminator.

For the DP case, images are captured by the two image sensorssimultaneously when the eye is illuminated by one of the one or more farimage sensors.

In addition to causing the BP and DP effect in the image used in orderto estimate the position of the representation of the pupil in theimage, the illuminator or illuminators used for capturing the images bymeans of the two image sensors also give rise to a respective glint usedfor determining gaze direction.

Depending on the position of each of the image sensors, each of theclose illuminators, and each of the one or more far illuminators, thepossibility to identify the glint in each of the two images will vary.In one example system the two image sensors and associated nearilluminators are arranged centrally in horizontal direction, and two farilluminators arranged one at one side and the other on the other sidehorizontally of the image sensors. Then, if a user, for which BP imagesare captured, is looking at a point far to one side of the centerhorizontally in relation to the two image sensors, glints produced bythe close illuminators may not be located on the best portion of thecornea for eye/gaze tracking. Similarly, if a user for which DP imagesare captured is looking at a point close to the center horizontally inrelation to the two image sensors, glints produced by the farilluminators may not be positioned on the best portion of the cornea forgaze tracking.

Hence, there are situations when identification of glints can becomedifficult or impossible, or identified glints may not be suitable to usefor estimating the gaze direction of the user, since the glints may notbe positioned on the best portion of the cornea for gaze tracking.

It would be desirable to provide further systems and methods addressingone or more of the issues with known systems and methods.

SUMMARY

Methods and systems are disclosed for use in controlling illuminators inan eye tracking system.

The eye tracking system includes a first image sensor/camera, a secondimage sensor/camera, a first close illuminator arranged to capturebright pupil (BP) images by the first image sensor, a second closeilluminator arranged to capture bright pupil (BP) images by the secondimage sensor and one or more far illuminators arranged to capture darkpupil images by the first image sensor and the second image sensor.

In example systems the first close illuminator is arranged coaxiallywith or near the first image sensor, the second close illuminator isarranged coaxially with or near the second image sensor and the one ormore far illuminators are arranged non-coaxially with or far from thefirst image sensor and the second image sensor.

According to example methods, two illuminators of the first closeilluminator, second close illuminator, and one or more far illuminators,are used for each exposure of each of the first image sensor and secondimage sensor. One of the illuminators used for a camera exposure is usedas a main illuminator and the other is used as a support illuminator.The main illuminator is selected to optimize or enhance pupil contrastand the support illuminator is selected to optimize or enhance glintposition on the cornea, i.e. a glint position close to the center of thecornea.

In a method for DP images, one of the one or more far illuminators (mainilluminator for the first image sensor) is maintained activated duringan exposure time of the first image sensor, and the same one of the oneor more far illuminators (main illuminator also for the second imagesensor) is also maintained activated during an exposure time of thesecond image sensor. Furthermore, the second close illuminator (supportilluminator for the first image sensor) is maintained activated during aportion of the exposure time of the first image sensor and the firstclose illuminator is maintained deactivated during the exposure time ofthe first image sensor. Similarly, the first close illuminator (supportilluminator for the second image sensor) is maintained activated duringa portion of the exposure time of the second image sensor, and thesecond close illuminator is maintained deactivated during the exposuretime of the second image sensor.

Selection of which of the one or more far illuminators should be used asa main illuminator for the first image sensor and the second imagesensor for a time frame (i.e. a period of time in which an image iscaptured for the first image sensor and the second image sensor,respectively) may be done based on eye tracking data from a previousframe. For example, if the eye tracking data from a previous frameindicates that the user is looking to the left, an illuminator of theone or more far illuminators being located to the left is selected as amain illuminator.

In a method for BP images, the first close illuminator is maintainedactivated (main illuminator for the first image sensor) during anexposure time of the first image sensor, and the second closeilluminator is maintained deactivated during the exposure time of thefirst image sensor. The second close illuminator (main illuminator forthe second image sensor) is maintained activated during an exposure timeof the second image sensor and the first close illuminator is maintaineddeactivated during the exposure time of the second image sensor.Furthermore, one of the one or more far illuminators (supportilluminator for the first image sensor) is maintained activated during aportion of the exposure time of the first image sensor, and the same oneof the one or more far illuminators (support illuminator also for thesecond image sensor) is maintained activated during a portion of theexposure time of the second image sensor. It is to be noted that thesupport illuminator for the second image sensor may in other examplemethods be a different one of the one or more far illuminators than thesupport illuminator for the first image sensor.

Selection of which of the one or more far illuminators should be used asa support illuminator for the first image sensor and the second imagesensor for a time frame (i.e. a period of time in which an image iscaptured for the first image sensor and the second image sensor,respectively) may be done based on eye tracking data from a previousframe. For example, if the eye tracking data from a previous frameindicates that the user is looking to the left, an illuminator of theone or more far illuminators being located to the left is selected as amain illuminator.

One aim is to minimize the total exposure time of the first image sensorand the second image sensor for each exposure cycle, i.e. to minimizethe time from the start of the exposure time of the image sensor of thefirst image sensor and the second image sensor for which the exposuretime starts first to the end of the exposure time of the image sensor ofthe first image sensor and the second image sensor for which theexposure time ends last.

For DP images, a main illuminator for the first image sensor is one ofthe one or more far illuminators arranged to capture DP images by thefirst image sensor, e.g. arranged non-coaxially with or far from thefirst image sensor. A main illuminator for the second image sensor issaid one of the one or more far illuminators arranged to capture DPimages by the second image sensor, e.g. arranged non-coaxially with orfar from the second image sensor.

Since the main illuminator for the first image sensor during exposureand the main illuminator for the second image sensor during exposure isthe same one of the one or more far illuminators, illumination of thefirst image sensor and the second image sensor during exposure by meansof said one of the one or more far illuminators can be concurrent.

For DP images, a support illuminator for the first image sensor is thesecond close illuminator arranged to capture BP images by the secondimage sensor, e.g. arranged coaxially with or near the second imagesensor. A support illuminator for the second image sensor is the firstclose illuminator arranged to capture BP images by the first imagesensor, e.g. arranged coaxially with or near the first image sensor.Illumination of the first image sensor during exposure by means of thefirst close illuminator will result in a BP effect and may thus resultin reduced contrast between a representation of a pupil and an iris,respectively, in the resulting image. Similarly, illumination of thesecond image sensor during exposure by means of the second closeilluminator will result in a BP effect and thus may result in reducedcontrast between a representation of a pupil and an iris, respectively,in the resulting image. Hence, the illumination of the first imagesensor during exposure by means of the second close illuminator, used asa support illuminator for the first sensor, has to be separated in timefrom exposure of the second image sensor. Similarly, the illumination ofthe second image sensor during exposure by means of the first closeilluminator, used as a support illuminator for the second sensor, has tobe separated in time from exposure of the first image sensor.

One way of limiting the time from the start of the exposure of the imagesensor of the first image sensor and the second image sensor for whichexposure starts first to the end of the exposure of the image sensor ofthe first image sensor and the second image sensor for which exposureends last for DP images, is to illuminate the first image sensor duringa first time period which is t_(main) long by means of said one of theone or more far illuminators and to illuminate the first image sensoralso by means of the second close illuminator during a second timeperiod which is t_(support) long at the beginning of the first timeperiod, where t_(support) is shorter than t_(main), and then after theend of the second time period illuminate the second image sensor duringa third time period which is t_(main) long by means of said one of theone of more far illuminators and to illuminate the second image sensoralso by means of the first close illuminator during a fourth time periodwhich is t_(support) long at the end of the third time period. By doingso, said one of the one or more far illuminators can be continuously litduring the first time period and the third time period, which are partlyoverlapping. Furthermore, the second close illuminator can be lit onlyduring the second time period which is the first t_(support) long partof the first time period and which does not overlap the third timeperiod during the exposure of the second image sensor. Hence, theillumination by means of the second close illuminator during the secondtime period will not affect the DP image of the second image sensor.Similarly, the first close illuminator can be lit only during the fourthtime period which is the last t_(support) long part of the third timeperiod and which does not overlap the first time period during theexposure of the first image sensor. Hence, the illumination by means ofthe first close illuminator during the fourth time period will notaffect the DP image of the first image sensor.

The length t_(support) of the second and fourth time period during whichthe second close illuminator and the first close illuminator,respectively, is lit should be long enough to produce a glint which canbe identified in the image but at the same time be short enough not toreduce the contrast between a representation of a pupil and an iris inthe image. In alternative or in combination with adapting the lengtht_(support) of the second and fourth time period, the power of thesecond close illuminator and the first close illuminator can be adaptedto be high enough to produce a glint which can be identified in theimage but at the same time be low enough not to reduce the contrastbetween a representation of a pupil and an iris in the image.

For BP images, a main illuminator for the first image sensor is thefirst close illuminator arranged to capture BP images by the first imagesensor, e.g. arranged coaxially with or near the first image sensor. Amain illuminator for the second image sensor is the second closeilluminator arranged to capture BP images by the second image sensor,e.g. arranged coaxially with or near the second image sensor.Illumination of the first image sensor during exposure by means of thesecond close illuminator may result in reduced contrast between arepresentation of a pupil and an iris, respectively, in the resultingimage. Similarly, illumination of the second image sensor duringexposure by means of the first close illuminator may result in reducedcontrast between a representation of a pupil and an iris, respectively,in the resulting image. As the first close illuminator and the secondclose illuminator are to be used as main illuminator for the first imagesensor and the second image sensor, respectively, during exposure, theexposure of the first image sensor and the exposure of the second imagesensor have to be separated in time.

For BP images, a support illuminator for the first image sensor is saidone of the one or more far illuminators arranged to capture DP images bythe first image sensor, e.g. arranged non-coaxially with or far from thefirst image sensor. A support illuminator for the second image sensor issaid one of the one or more illuminators arranged to capture DP imagesby the second image sensor, e.g. arranged non-coaxially with or far fromthe second image sensor. Illumination of the first image sensor duringexposure by means of said one of the one or more far illuminatorsgenerally does not result in substantially reduced contrast between arepresentation of a pupil and an iris, respectively, in the resultingimage. Similarly, illumination of the second image sensor duringexposure by means of said one of the one or more far illuminatorsgenerally does not result in substantially reduced contrast between arepresentation of a pupil and an iris, respectively, in the resultingimage. Hence, illumination of the first image sensor during exposure bymeans of said one of the one or more far illuminators can be concurrentwith illumination of the first image sensor by means of the first closeilluminator. Similarly, illumination of the second image sensor duringexposure by means of said one of the one or more far illuminators can beconcurrent with illumination of the second image sensor by means of thesecond close illuminator.

One way of limiting the time from the start of the exposure time of theimage sensor of the first image sensor and the second image sensor forwhich the exposure time starts first to the end of the exposure time ofthe image sensor of the first image sensor and the second image sensorfor which the exposure time ends last for BP images, is to illuminatethe first image sensor during a first time period which is t_(main) longby means of the first close illuminator and to illuminate the firstimage sensor also by means of said one of the one or more farilluminators during a second time period which is t_(support) long atthe end of the first time period, where t_(support) is shorter thant_(main), and then after the end of the first time period illuminate thesecond image sensor during a third time period which is t_(main) long bymeans of the second close illuminator and to illuminate the second imagesensor also by means of said one of the one or more far illuminatorsduring a fourth time period which is t_(support) long at the beginningof the third time period. By doing so, said one of the one or more farilluminators can be continuously lit during the final t_(support) longpart of the first time period and the first t_(support) long part of thethird time period. In alternative the time that said one of the one ormore far illuminators is lit up can be divided into two periods whichoverlap any part of the first time period and the third time period,respectively.

The length t_(support) of the second and fourth time period during whichsaid one of the one or more far illuminators is lit should be longenough to produce a glint which can be identified in the image but atthe same time be short enough not to reduce the contrast between arepresentation of a pupil and an iris in the image. In alternative or incombination with adapting the length t_(support) of the second andfourth time period, the power of said one of the one or more farilluminators can be adapted to be high enough to produce a glint whichcan be identified in the image but at the same time be low enough not toreduce the contrast between a representation of a pupil and an iris inthe image.

BRIEF DESCRIPTION OF THE DRAWINGS

Exemplifying embodiments will be described below with reference to theaccompanying drawings, in which:

FIG. 1 shows an example of an eye tracking system in which embodimentscan be implemented;

FIG. 2 shows another example of an eye tracking system in whichembodiments can be implemented;

FIG. 3 shows an example DP image of an eye;

FIG. 4 shows an example BP image of an eye;

FIG. 5a shows a DP image of an eye with one peripherally positionedglint using a main illuminator only;

FIG. 5b shows a DP image of an eye with one peripherally positionedglint and one centrally positioned glint using a main illuminator and asupport illuminator;

FIG. 6a shows a BP image of an eye with one peripherally positionedglint using a main illuminator only;

FIG. 6b shows a BP image of an eye with one peripherally positionedglint and one centrally positioned glint using a main illuminator and asupport illuminator;

FIG. 7a shows a timing diagram for camera exposure and illuminators forDP images;

FIGS. 7b-c show timing diagrams for camera exposure and illuminators forBP images; and

FIGS. 8-9 are flow charts of methods according to embodiments.

All the figures are schematic and generally only show parts which arenecessary in order to elucidate the respective embodiments, whereasother parts may be omitted or merely suggested.

DESCRIPTION

FIG. 1 shows an example of an eye/gaze tracking system 100, in whichembodiments may be implemented. The system 100 comprises illuminators111-114 for illuminating the eyes of a user, and light sensors 115, 116for capturing images of the eyes of the user. The illuminators 111-114may for example be light emitting diodes emitting light in the infraredfrequency band, or in the near infrared frequency band. The lightsensors 115, 116 may for example be an image sensor, such as acomplementary metal oxide semiconductor (CMOS) image sensor or a chargedcoupled device (CCD) image sensor. The image sensor may consist of anintegrated circuit containing an array of pixel sensors, each pixelcontaining a photodetector and an active amplifier. The image sensor iscapable of converting light into digital signals. In reality, as anexample, it could be

-   -   Infrared image sensor or IR image sensor    -   RGB sensor    -   RGBW sensor    -   RGB or RGBW sensor with IR filter

First illuminators 111, 112, close illuminators 111, 112 herein, arearranged coaxially with (or close to) the light sensors 115, 116,respectively, so that the light sensors 115, 116 may capture brightpupil images of the eyes of a user (subject). Due to the coaxialarrangement of the close illuminators 111, 112 and the light sensors115, 116, respectively, light reflected from the retina of an eyereturns back out through the pupil towards the light sensors 115, 116,so that the pupil appears brighter than the iris surrounding it inimages where the close illuminators 111, 112 illuminate the eye. Secondilluminators 113, 114, far illuminators 113, 114 herein, are arrangednon-coaxially with (or further away from) the light sensors 115, 116 forcapturing dark pupil images. Due to the non-coaxial arrangement of thefar illuminators 113, 114 and the light sensors 115, 116 light reflectedfrom the retina of an eye does not reach the light sensors 115, 116 andthe pupil appears darker than the iris surrounding it in images wherethe far illuminators 113, 114 illuminate the eye. The close illuminators111, 112 may for example be used to illuminate the eye of a user if theuser has a good/high BP response, i.e. high contrast betweenrepresentations of pupil and iris in BP images, and the far illuminators113, 114 may for example be used to illuminate the eye of a user if theuser has a bad/low BP response, i.e. low contrast betweenrepresentations of pupil and iris in BP images.

The eye tracking system 100 also comprises circuitry 120 (for exampleincluding one or more processors) for processing the images captured bythe light sensors 115, 116. The circuitry 120 may for example beconnected to the light sensors 115, 116 and the illuminators 111-114 viaa wired or a wireless connection. In another example, circuitry 120 inthe form of one or more processors may be provided in one or morestacked layers below the light sensitive surface of the light sensors115, 116.

FIG. 2 shows another example of an eye/gaze tracking system 200, inwhich embodiments may be implemented. First illuminators 211, 212, closeilluminators 211, 212 herein, are arranged coaxially with (or close to)the light sensors 215, 216, respectively, so that the light sensors 215,216 may capture BP images of the user's eyes. A further illuminator 213,far illuminator 213 herein, is arranged non-coaxially with (or furtheraway from) the light sensors 215, 216 for capturing DP images. The eyetracking system 200 also comprises circuitry 220 (for example includingone or more processors) for processing the images captured by the lightsensors 215, 216.

In the systems described with reference to FIG. 1 and FIG. 2, theilluminators 111-114; 211-213 are arranged in an eye tracking module110; 210 placed below a display watched by the user. This arrangementserves only as an example. It will be appreciated that more illuminatorsand more light sensors may be employed for eye tracking, and that suchilluminators and light sensors may be distributed in many different waysrelative to displays watched by the user. For example, further farilluminators may be provided to be used as support illuminators for BPimages and as support or main illuminators for DP images. Such furtherilluminators may be positioned in order to take account for further gazedirections of the user to produce glints that are positioned close tothe center of the cornea. For example, further far illuminators can bepositioned above a display to take account for gaze directions closer tothe top of the display.

Furthermore, each of the close illuminators 111, 112; 211, 212 in FIG. 1and FIG. 2, respectively, may be located as a circle around a respectiveone of the light sensors 115, 116; 215, 216, respectively, to be trulycoaxially arranged.

FIG. 3 shows an example of a first DP image of an eye 300, captured byone of the light sensors 115, 116; 215, 216. DP images are used in orderto achieve a best contrast as possible between a representation of apupil 310 and an iris 320 of the eye, respectively, for users with lowBP response. A second image (not shown) is captured by the other one ofthe light sensors 115, 116; 215, 216. The circuitry 120; 220 may forexample employ image processing (such as digital image processing) forextracting features in the first image (and the second image). Thecircuitry 120; 220 may for example employ pupil center cornea reflection(PCCR) eye tracking to determine a position of the eye 300 in space anda point of gaze or gaze direction of the eye 300. In PCCR eye tracking,using one of the systems of FIG. 1 and FIG. 2, the position of the eye300 in space and the point of gaze or gaze direction of the eye 300 isdetermined based on an estimation of a position of a center of therepresentation of the pupil 310, a position of a center of a glint 330at the eye 300 in the first image, and corresponding estimations of aposition of a center of a pupil, a position of a center of a glint atthe eye in the second image.

FIG. 4 shows an example of a first BP image of an eye 300, captured byone of the light sensors 115, 116; 215, 216. BP images are used in orderto achieve a best contrast as possible between a representation of apupil 410 and an iris 420 of the eye, respectively, for users with highBP response. A second image (not shown) is captured by the other one ofthe light sensors 115, 116; 215, 216. The circuitry 120; 220 may forexample employ image processing (such as digital image processing) forextracting features in the first image (and the second image). Thecircuitry 120; 220 may for example employ pupil center cornea reflection(PCCR) eye tracking to determine a position of the eye 400 in space anda point of gaze or gaze direction of the eye 400. In PCCR eye tracking,using one of the systems of FIG. 1 and FIG. 2, the position of the eye400 in space and the point of gaze or gaze direction of the eye 400 isdetermined based on an estimation of a position of a center of therepresentation of the pupil 410, a position of a center of a glint 430at the eye 300 in the first image, and corresponding estimations of aposition of a center of a pupil, a position of a center of a glint atthe eye in the second image.

FIG. 5a shows a DP image of an eye with one peripherally positionedglint using a main illuminator only. When only a main illuminator is litduring exposure of a light sensor, such as the rightmost far illuminator114 for the leftmost image sensor 115 in the system 100 shown in FIG. 1,the DP image of an eye of a user having a central gaze direction suchthat the pupil is centered in the image of the eye, a glint 530 in theDP image may be located far to the left in the eye in relation to therepresentation of the pupil 510 in the image. This may correspond to theglint being located outside the central region of the cornea which isapproximately spherical which in turn will produce less accurate resultsin relation to eye/gaze tracking.

FIG. 5b shows a DP image of an eye with one peripherally positionedglint and one centrally positioned glint using a main illuminator and asupport illuminator. In addition to a main illuminator lit duringexposure of a light sensor, such as the rightmost far illuminator 114for the leftmost image sensor 115 in the system 100 shown in FIG. 1, afurther support illuminator may be lit during exposure of the lightsensor 115, such as the close illuminator 112 associated with the otherimages sensor 116 of the system 100 in FIG. 1. By using the closeilluminator 112 of the other image sensor 116, no BP effect will becaused in the image of the light sensor 115. Furthermore, since theclose illuminator 112 of the other image sensor 116 is centrallypositioned in the system 100, the resulting glint 540 will be morecentrally positioned in relation to the representation of the pupil 510in an image of an eye of a user having a central gaze direction suchthat the pupil is centered in the image of the eye. This may correspondto the glint being located inside the central region of the cornea whichis approximately spherical which in turn will produce more accurateresults in relation to eye/gaze tracking.

FIG. 6a shows a BP image of an eye with one peripherally positionedglint using a main illuminator only. When only a main illuminator is litduring exposure of a light sensor, such as the close illuminator 111associated with the leftmost image sensor 115 in the system 100 shown inFIG. 1, the BP image of an eye of a user having a gaze direction far tothe right such that the pupil is positioned far to the left in the imageof the eye, a glint 630 in the DP image may be located far to the rightin the eye in relation to the representation of the pupil 610 in theimage. This may correspond to the glint being located outside thecentral region of the cornea which is approximately spherical which inturn will produce less accurate results in relation to eye/gazetracking.

FIG. 6b shows a BP image of an eye with one peripherally positionedglint and one centrally positioned glint using a main illuminator and asupport illuminator. In addition to a main illuminator lit duringexposure of a light sensor, such as the close illuminator 111 associatedwith the leftmost image sensor 115 in the system 100 shown in FIG. 1, afurther support illuminator may be lit during exposure of the lightsensor 115, such as the rightmost far illuminator 114 of the system 100in FIG. 1. By using the rightmost far illuminator 114, the resultingglint 640 will be more centrally positioned in relation to therepresentation of the pupil in an image of an eye of a user having agaze direction far to the right such that the pupil is located far tothe left in the image of the eye. This may correspond to the glint beinglocated inside the central region of the cornea which is approximatelyspherical which in turn will produce more accurate results in relationto eye/gaze tracking.

FIG. 7a shows a timing diagram for camera exposure and illuminators forDP images. A first image sensor C₁ and a second image sensor C₂ are eachexposed during a time period which is t_(main) long. Exposure isindicated as a high in the diagram associated to each of the first imagesensor C₁ and the second image sensor C₂, respectively. During theexposure of the first image sensor C₁ and the exposure of the secondimage sensor C₂ a main illuminator I_(p) is lit.

The main illuminator I_(p) for the first image sensor C₁ and the secondimage sensor C₂ is an illuminator arranged to capture DP images by therespective image sensor, e.g. arranged non-coaxially with or far fromthe respective sensor, (such as the leftmost far illuminator 113 or therightmost far illuminator 114 of the system 100 of FIG. 1). Since themain illuminator I_(p) for both image sensors during exposure is thesame, illumination of the first image sensor C₁ and the second imagesensor C₂ during exposure by means of the main illuminator I_(p) can beconcurrent as indicated in the diagram related to the main illuminatorI_(p) in FIG. 7 a.

Generally, in relation to the system 100 of FIG. 1, the leftmost farilluminator 113 would be used as the main illuminator I_(p) if the useris looking to the left and the rightmost far illuminator 114 is used asthe main illuminator I_(p) if the user is looking to the right.Furthermore, selection of which of the one or more far illuminators 113,114 should be used as a main illuminator for the first image sensor 115and the second image sensor 116 for a time frame (i.e. a period of timein which an image is captured for the first image sensor and the secondimage sensor, respectively) may be done based on eye tracking data froma previous frame. For example, if the eye tracking data from a previousframe indicates that the user is looking to the right, an illuminator114 of the one or more far illuminators 113, 114 being located to theright is selected as a main illuminator.

For DP images, a support illuminator I₂ for the first image sensor C₁ isan illuminator arranged to capture BP images by the second image sensorC₂, e.g. arranged coaxially with or near the second image sensor C₂(such as the close illuminator 112 associated with the second imagesensor 116 of the system 100 of FIG. 1). Furthermore, a supportilluminator I₁ for the second image sensor C₂ is an illuminator arrangedto capture BP images by the first image sensor C₁, e.g. arrangedcoaxially with or near the first image sensor C₁ (such as the closeilluminator 111 associated with the first image sensor 115 of the system100 of FIG. 1). Illumination of the first image sensor C₁ duringexposure by means of the close illuminator associated with the firstimage sensor C₁ and used as a support illuminator for the second imagesensor C₂ will result in a BP effect and may thus result in reducedcontrast between a representation of a pupil and an iris, respectively,in the resulting image. Similarly, illumination of the second imagesensor C₂ during exposure by means of the close illuminator associatedwith the second image sensor C₂ and used as a support illuminator forthe first image sensor C₁ will result in a BP effect and thus may resultin reduced contrast between a representation of a pupil and an iris,respectively, in the resulting image. Hence, the illumination of thefirst image sensor C₁ during exposure by means of the close illuminatorassociated with the second image sensor C₂, used as a supportilluminator I₂ for the first image sensor C₁, has to be separated intime from exposure of the second image sensor C₂. Similarly, theillumination of the second image sensor C₂ during exposure by means ofthe close illuminator associated with the first image sensor C₁, used asa support illuminator I₁ for the second image sensor C₂, has to beseparated in time from exposure of the first image sensor C₁.

One way of limiting the time from the start of the exposure of the imagesensor of the first image sensor and the second image sensor for whichexposure starts first to the end of the exposure of the image sensor ofthe first image sensor and the second image sensor for which exposureends last for DP images, is shown in the time diagram of FIG. 7a . Thefirst image sensor C₁ is illuminated a first time period which ist_(main) long by means of the main illuminator I_(p). During a secondtime period which is t_(support) long at the beginning of andoverlapping the first time period, where t_(support) is shorter thant_(main), the first image sensor C₁ is illuminated also by the supportilluminator I₂. At the end of the second time period, i.e. t_(support)from the start of the exposure of the first image sensor C₁, the secondimage sensor C₂ is illuminated during a third time period which ist_(main) long by means of the main illuminator I_(p). During a fourthtime period which is t_(support) long at the end of and overlapping thethird time period and starting at the end of the first time period, thesecond image sensor C₂ is illuminated also by the support illuminatorI₁.

By doing so, the support illuminator I₂ for the first image sensor C₁which is the close illuminator for the second image sensor C₂, is notlit during the third time period, i.e. the exposure time of the secondimage sensor C₂. Similarly, the support illuminator I₁ for the secondimage sensor C₂ which is the close illuminator for the first imagesensor C₁, is not lit during the first time period, i.e. the exposuretime of the first image sensor C₁. The total time from the start of theexposure of the first image sensor C₁ to the end of the exposure of thesecond image sensor C₂ is t_(main) t_(support) long.

The length t_(support) of the time periods during which the supportilluminator I₂ of the first image sensor C₁ and the support illuminatorI₁ of the second image sensor C₂, respectively, are lit, should be longenough to produce a glint which can be identified in the image but atthe same time be short enough not to reduce the contrast between arepresentation of a pupil and an iris in the image. In alternative or incombination with adapting the length t_(support) of the second andfourth time period, the power of the support illuminator I₂ for thefirst image sensor C₁ and the support illuminator I₁ for the secondimage sensor C₂ can be adapted to be high enough to produce a glintwhich can be identified in the image but at the same time be low enoughnot to reduce the contrast between a representation of a pupil and aniris in the image.

FIG. 7b shows a timing diagram for camera exposure and illuminators forBP images. A first image sensor C₁ and a second image sensor C₂ are eachexposed during a time period which is t_(main) long. Exposure isindicated as a high in the diagram associated to each of the first imagesensor C₁ and the second image sensor C₂, respectively. During theexposure of the first image sensor C₁ a main illuminator I₁ is lit andduring the exposure of the second image sensor C₂ a main illuminator I₂is lit.

The main illuminator I₁ for the first image sensor C₁ is an illuminatorarranged to capture BP images by the first image sensor C₁, e.g.arranged coaxially with or close to the first image sensor C₁ (such asthe close illuminator 111 associated with the leftmost image sensor 115of the system 100 of FIG. 1). The main illuminator I₂ for the secondimage sensor C₂ is an illuminator arranged to capture BP images by thesecond image sensor C₂, e.g. arranged coaxially with or close to therespective sensor (such as the close illuminator 112 associated with thesecond image sensor C₂ of the system 100 of FIG. 1).

The support illuminator I_(p) for the first image sensor C₁ and thesecond image sensor C₂ is an illuminator arranged to capture DP imagesby the respective image sensor, e.g. arranged non-coaxially with or farfrom the respective sensor, (such as the leftmost far illuminator 113 orthe rightmost far illuminator 114 of the system 100 of FIG. 1).

Generally, in relation to the system 100 of FIG. 1, the leftmost farilluminator 113 would be used as the support illuminator I_(p) if theuser is looking to the left and the rightmost far illuminator 114 isused as the support illuminator I_(p) if the user is looking to theright. Furthermore, selection of which of the far illuminators 113, 114to be used as a support illuminator for the first image sensor 115 andthe second image sensor 116 for a time frame (i.e. a period of time inwhich an image is captured for the first image sensor and the secondimage sensor, respectively) may be done based on eye tracking data froma previous frame. For example, if the eye tracking data from a previousframe indicates that the user is looking to the left, an illuminator 113of the one or more far illuminators 113, 114 being located to the leftis selected as a support illuminator.

Illumination of the first image sensor C₁ during exposure by means ofthe second close illuminator I₂ may result in a reduced contrast betweena representation of a pupil and an iris, respectively, in the resultingimage. Similarly, illumination of the second image sensor C₂ duringexposure by means of the first close illuminator I₁ may result in areduced contrast between a representation of a pupil and an iris,respectively, in the resulting image. As the first close illuminator I₁and the second close illuminator I₂ are to be used as main illuminatorfor the first image sensor C₁ and the second image sensor C₂,respectively, during exposure, the exposure of the first image sensorand the exposure of the second image sensor have to be separated intime.

One way of limiting the time from the start of the exposure of the imagesensor of the first image sensor and the second image sensor for whichexposure starts first to the end of the exposure of the image sensor ofthe first image sensor and the second image sensor for which exposureends last for BP images, is shown in the time diagram of FIG. 7b . Thefirst image sensor C₁ is illuminated a first time period which ist_(main) long by means of the main illuminator I₁ associated with thefirst image sensor C₁. During a second time period which is t_(support)long at the end of and overlapping the first time period, wheret_(support) is shorter than t_(main), the first image sensor C₁ isilluminated also by the support illuminator I_(p). The end of the secondtime period coincides with the end of the first time period and ist_(main) from the start of the exposure of the first image sensor C₁. Atthe end of the first time period, the second image sensor C₂ isilluminated during a third time period which is t_(main) long by meansof the main illuminator I₂ associated with the first image sensor C₂.During a fourth time period which is t_(support) long at the beginningof and overlapping the third time period and starting at the end of thefirst time period, the second image sensor C₂ is illuminated also by thesupport illuminator I_(p).

By doing so, the main illuminator I₁ for the first image sensor C₁ isnot lit during the third time period, i.e. the exposure time of thesecond image sensor C₂. Similarly, the main illuminator I₂ for thesecond image sensor C₂ is not lit during the first time period, i.e. theexposure time of the first image sensor C₁. The total time from thestart of the exposure of the first image sensor C₁ to the end of theexposure of the second image sensor C₂ is t_(main)+t_(main) long.

The length t_(support) of the time periods during which the supportilluminator I_(p) is lit for the first image sensor C₁ and the supportilluminator I_(p) is lit for the second image sensor C₂, respectively,should be long enough to produce a glint which can be identified in theimage but at the same time be short enough not to reduce the contrastbetween a representation of a pupil and an iris in the image. Inalternative or in combination with adapting the length t_(support) ofthe second and fourth time period, the power of the support illuminatorI_(p) can be adapted to be high enough to produce a glint which can beidentified in the image but at the same time be low enough not to reducethe contrast between a representation of a pupil and an iris in theimage.

FIG. 7c shows a timing diagram for camera exposure and illuminators forBP images similar to the timing diagram in FIG. 7b . The timing for theexposure of the first image sensor C₁, the main illuminator I₁ for thefirst image sensor C₁, the exposure of the second image sensor I₂, andthe main illuminator I₂ for the second image sensor C₂ is the same as inFIG. 7b . However, the timing for the support illuminator I_(p) for thefirst image sensor C₁ and for the second image sensor C₂ differs in thatthe second time period during which the support illuminator I_(p) is litfor the first image sensor C₁ starts at the beginning of the first timeperiod, i.e. at the start of the exposure of the first image sensor C₁and the fourth time period during which the support illuminator I_(p) islit for the second image sensor C₂ ends at the end of the third timeperiod, i.e. at the end of the exposure of the first image sensor C₂.

FIG. 8 shows a flow chart of a method in an eye tracking system. Thesystem comprises a first image sensor, a second image sensor, a firstclose illuminator arranged to capture bright pupil images by the firstimage sensor, a second close illuminator arranged to capture brightpupil images by the second image sensor and one or more far illuminatorsarranged to capture dark pupil images by the first image sensor and thesecond image sensor. In the method one of the one or more farilluminators is maintained 810 activated during an exposure time of thefirst image sensor. The second close illuminator is maintained 820activated during a portion of the exposure time of the first imagesensor and the first close illuminator is maintained 830 deactivatedduring the exposure time of the first image sensor. Said one of the oneor more far illuminators is also maintained 840 activated during anexposure time of the second image sensor. Furthermore, the first closeilluminator is maintained 850 activated during a portion of the exposuretime of the second image sensor, and the second close illuminator ismaintained 860 deactivated during the exposure time of the second imagesensor.

FIG. 9 shows a flow chart of a method in an eye tracking system. Thesystem comprises a first image sensor, a second image sensor, a firstclose illuminator arranged to capture bright pupil images by the firstimage sensor, a second close illuminator arranged to capture brightpupil images by the second image sensor and one or more far illuminatorsarranged to capture dark pupil images by the first image sensor and thesecond image sensor. In the method the first close illuminator ismaintained 910 activated during an exposure time of the first imagesensor, and the second close illuminator is maintained 920 deactivatedduring the exposure time of the first image sensor. Furthermore, one ofthe one or more far illuminators is maintained 930 activated during aportion of the exposure time of the first image sensor. The second closeilluminator is maintained 940 activated during an exposure time of thesecond image sensor and the first close illuminator is maintained 950deactivated during the exposure time of the second image sensor.Furthermore, said one of the one or more far illuminators is maintained960 activated during a portion of the exposure time of the second imagesensor.

It is to be noted that the steps in FIGS. 8 and 9 relates to maintainingof illuminators activated during a period of time. The order of thesteps is not intended to illustrate a sequential order of the steps.Instead, some of the steps may be performed in parallel to each other.

A person skilled in the art realizes that the present invention is by nomeans limited to the preferred embodiments described above. On thecontrary, many modifications and variations are possible within thescope of the appended claims. For example, the person skilled in the artrealizes that the methods described herein may be performed by manyother eye/gaze tracking systems than the example eye/gaze trackingsystems 100 and 200 shown in FIGS. 1 and 2. Furthermore, methods andsystems have been described for one eye. However, a person skilled inthe art realizes that the methods may be performed for two eyes alsowhere the result for both eyes is taken into account in each step.

Additionally, variations to the disclosed embodiments can be understoodand effected by those skilled in the art in practicing the claimedinvention, from a study of the drawings, the disclosure, and theappended claims. In the claims, the word “comprising” does not excludeother elements or steps, and the indefinite article “a” or “an” does notexclude a plurality. The division of tasks between functional unitsreferred to in the present disclosure does not necessarily correspond tothe division into physical units; to the contrary, one physicalcomponent may have multiple functionalities, and one task may be carriedout in a distributed fashion, by several physical components incooperation. A computer program may be stored/distributed on a suitablenon-transitory medium, such as an optical storage medium or asolid-state medium supplied together with or as part of other hardware,but may also be distributed in other forms, such as via the Internet orother wired or wireless telecommunication systems. The mere fact thatcertain measures/features are recited in mutually different dependentclaims does not indicate that a combination of these measures/featurescannot be used to advantage. Method steps need not necessarily beperformed in the order in which they appear in the claims or in theembodiments described herein, unless it is explicitly described that acertain order is required. Any reference signs in the claims should notbe construed as limiting the scope.

The invention claimed is:
 1. A method of controlling illuminators in an eye tracking system, the method comprising: activating a main illuminator and a support illuminator during an exposure time of a first image sensor, wherein the main illuminator and the support illuminator are in an eye tracking system and wherein the main illuminator is one of a first close illuminator a second close illuminator, or one or more far illuminators, and wherein the support illuminator is one of the first close illuminator, the second close illuminator, or the one or more far illuminators and wherein the first close illuminator and second close illuminator are to illuminate an eye for bright pupil image capture by an image sensor and the far illuminators are to illuminate the eye for dark pupil image capture; maintaining one of the first close illuminator or the second close illuminator deactivated during the exposure time of the first image sensor; activating the main illuminator and the support illuminator during an exposure time of a second image sensor, wherein the main illuminator is one of the first close illuminator, the second close illuminator, or the one or more far illuminators, and wherein the support illuminator is one of the first close illuminator, the second close illuminator, or the one or more far illuminators; maintaining one of the first close illuminator or the second close illuminator deactivated during the exposure time of the second image sensor; wherein the main illuminator during the exposure time of the first image sensor is the first close illuminator to capture a bright pupil image; the support illuminator during the exposure time of the first image sensor is the one or more of the far illuminators to capture a dark pupil image; the main illuminator during the exposure time of the second image sensor is the second close illuminator to capture a bright pupil image; and the support illuminator during the exposure time of the second image sensor is the one or more of the far illuminators to capture dark pupil images.
 2. The method of claim 1, further comprising: the main illuminator during the exposure time of the first image sensor is the one or more of the far illuminators; the support illuminator during the exposure time of the first image sensor is the second close illuminator; the main illuminator during the exposure time of the second image sensor is the one or more of the far illuminators; and the support illuminator during the exposure time of the second image sensor is the first close illuminator.
 3. The method of claim 1, wherein the first close illuminator is arranged coaxially with or near the first image sensor, the second close illuminator is arranged coaxially or near the second image sensor and the one or more far illuminators are arranged non-coaxially with or far from the first image sensor and the second image sensor.
 4. The method of claim 3, wherein the first image sensor and the second image sensor are arranged centrally in a horizontal direction in the system, and wherein the one or more far illuminators include a first far illuminator arranged to the left horizontally from the first image sensor and the second image sensor, and a second far illuminator arranged to the right horizontally from the first image sensor and the second image sensor.
 5. The method of claim 4, further comprising: activating the first far illuminator during the exposure time of the first image sensor and during the exposure time of the second image sensor on a condition that a gaze direction is estimated to be to the left; and activating the second far illuminator during the exposure time of the first image sensor and during the exposure time of the second image sensor on a condition that a gaze direction is estimated to be to the right.
 6. The method of claim 4, further comprising: activating the first far illuminator during a portion of the exposure time of the first image sensor on a condition that a gaze direction is estimated to be to the left; and activating the second far illuminator during a portion of the exposure time of the first image sensor on a condition that a gaze direction is estimated to be to the right; activating the first far illuminator during a portion of the exposure time of the second image sensor on a condition that a gaze direction is estimated to be to the left; and activating the second far illuminator during a portion of the exposure time of the second image sensor on a condition that a gaze direction is estimated to be to the right.
 7. The method of claim 1, further comprising: activating one of the one or more far illuminators based upon eye tracking data from a previous frame during the exposure time of the first image sensor; and activating one of the one or more far illuminators based upon eye tracking data from a previous frame during the exposure time of the second image sensor.
 8. An eye tracking system comprising: a first image sensor; a second image sensor; a first close illuminator; a second close illuminator; one or more far illuminators; and circuitry configured to: activating a main illuminator and a support illuminator during an exposure time of the first image sensor, wherein the main illuminator is one of the first close illuminator, the second close illuminator, or the one or more far illuminators, and wherein the support illuminator is one of the first close illuminator, the second close illuminator, or the one or more far illuminators and wherein the first close illuminator and second close illuminator are to illuminate an eye for bright pupil image capture by an image sensor and the far illuminators are to illuminate the eye for dark pupil image capture; maintaining one of the first close illuminator or the second close illuminator deactivated during the exposure time of the first image sensor; activating the main illuminator and the support illuminator during an exposure time of the second image sensor, wherein the main illuminator is one of the first close illuminator, the second close illuminator, or the one or more far illuminators, and wherein the support illuminator is one of the first close illuminator, the second close illuminator, or the one or more far illuminators; maintaining one of the first close illuminator or the second close illuminator deactivated during the exposure time of the second image sensor; wherein the main illuminator during the exposure time of the first image sensor is the first close illuminator to capture a bright pupil image; the support illuminator during the exposure time of the first image sensor is the one or more of the far illuminators to capture a dark pupil image; the main illuminator during the exposure time of the second image sensor is the second close illuminator to capture a bright pupil image; and the support illuminator during the exposure time of the second image sensor is the one or more of the far illuminators to capture dark pupil images.
 9. The eye tracking system of claim 8, further comprising: the main illuminator during the exposure time of the first image sensor is the one or more of the far illuminators; the support illuminator during the exposure time of the first image sensor is the second close illuminator; the main illuminator during the exposure time of the second image sensor is the one or more of the far illuminators; and the support illuminator during the exposure time of the second image sensor is the first close illuminator.
 10. The eye tracking system of claim 8, further comprising: activating one of the one or more far illuminators based upon eye tracking data from a previous frame during the exposure time of the first image sensor; and activating one of the one or more far illuminators based upon eye tracking data from a previous frame during the exposure time of the second image sensor.
 11. A non-transitory machine readable medium having instructions, the instructions executable by one or more processors to perform operations comprising: activating a main illuminator and a support illuminator during an exposure time of a first image sensor, wherein the main illuminator and the support illuminator are in an eye tracking system and wherein the main illuminator is one of a first close illuminator a second close illuminator, or one or more far illuminators, and wherein the support illuminator is one of the first close illuminator, the second close illuminator, or the one or more far illuminators and wherein the first close illuminator and second close illuminator are to illuminate an eye for bright pupil image capture by an image sensor and the far illuminators are to illuminate the eye for dark pupil image capture; maintaining one of the first close illuminator or the second close illuminator deactivated during the exposure time of the first image sensor; activating the main illuminator and the support illuminator during an exposure time of the second image sensor, wherein the main illuminator is one of the first close illuminator, the second close illuminator, or the one or more far illuminators, and wherein the support illuminator is one of the first close illuminator, the second close illuminator, or the one or more far illuminators; maintaining one of the first close illuminator or the second close illuminator deactivated during the exposure time of the second image sensor; wherein the main illuminator during the exposure time of the first image sensor is the first close illuminator to capture a bright pupil image; the support illuminator during the exposure time of the first image sensor is the one or more of the far illuminators to capture a dark pupil image; the main illuminator during the exposure time of the second image sensor is the second close illuminator to capture a bright pupil image; and the support illuminator during the exposure time of the second image sensor is the one or more of the far illuminators to capture dark pupil images.
 12. The system of claim 11, further comprising: the main illuminator during the exposure time of the first image sensor is the one or more of the far illuminators; the support illuminator during the exposure time of the first image sensor is the second close illuminator; the main illuminator during the exposure time of the second image sensor is the one or more of the far illuminators; and the support illuminator during the exposure time of the second image sensor is the first close illuminator.
 13. The system of claim 11, further comprising: activating one of the one or more far illuminators based upon eye tracking data from a previous frame during the exposure time of the first image sensor; and activating one of the one or more far illuminators based upon eye tracking data from a previous frame during the exposure time of the second image sensor. 